Matching and recommending relevant videos and media to individual search engine results

ABSTRACT

A computer-implemented system and process for generating video search engine results page is disclosed. The system provides a query term and retrieves a collection of search results. Tags are generated for each search result and used to match media objects to each search result. The search results and video objects related to each search result are returned as a video search engine results page.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.11/936,738, filed Nov. 7, 2007, now U.S. Pat. No. 8,037,051 which claimsthe benefit of priority to U.S. Provisional Application No. 60/857,706,filed on Nov. 8, 2006, both of which are hereby incorporated byreference in their entireties.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to information storage and retrievalsystems. More particularly, the present invention relates to a systemfor retrieving media objects related to search engine results andproviding those results to a user.

2. Description of the Related Art

Phrase based or keyword searching is a common method of searching usedfor electronic data. Keyword searching searches throughout aninformation database for instances of the words in the search query.Keyword searching does not, however, give results based on relevance;search query results often include items with no relevance orrelationship to one another other than the instance of a word in thesearch query. For example, a user intending to search products by thetechnology company Apple may enter the search query “Apple.” The searchresults, however, would likely include items relating to the applefruit, songs by the music label Apple, and so on. Consequently, thesearch query results of phrase based searching often have nothing incommon with the user's search intent.

Search methods which relate one object to another object are often usedin place of keyword searching in order to provide search query resultsrelevant to the searcher's intent. Such relationship-based searchmethods vary widely and range from precise to general catch-allapproaches. Methods relating text objects can vary widely in precisionand approach, quality and quantity. For example, Caid et al., in U.S.Pat. No. 5,619,709, titled “System and Method of Context VectorGeneration and Retrieval” relies on context vector generations and datedneural network approaches as opposed to more advanced auto-associativeapproaches. Weissman et al, in U.S. Pat. No. 6,816,857, uses methods ofdistance calculation to determine relationships for the purpose ofplacing meaning-based advertising on websites or to rate documentrelevance in currently used search engines.

These relationship based searches do not, however simulate the processthat a human would use in analyzing relevant information to relateobjects with one another. Starting with an object of interest, aresearcher typically researches within certain contexts and formsrelationships between information gathered during the process of readingand analyzing literature. During this flexible process, the context ofinterest may change, become refined or shift and take on a new directiondepending on the information found or thought processes of theresearcher. After the researcher finishes the research process, he isleft with a valuable collection of information that is related to aspecific theme or context of interest. For example, if the researcher'sobject of interest was a period of music and the context was the Baroquestyle, then a researcher might relate compositions to one another,compositions to a composer, compositions to a geographical location ortime period. Common relationship-based searches do not simulate thisprocess because they are both inflexible and non-interactive; theyneither allow a user to define and control the context and individualrelationships during the search, nor do they allow for the quality andquantity of relationships to be determined and visualized interactivelyby the user.

The results of these searches may not identify relevant portions ofretrieved documents or the relevance of an entire database. For example,keyword searching may identify portions of a document in which a term isused in the wrong context. Such systems do not allow a user to quicklyfind and understand the most relevant portions of a document and therelationship of that document to the user's search. The user may berequired to dig through large amounts of materials for an extendedperiod of time to identify these sections.

Furthermore, these systems do not identify materials and media relatedto the search results that a more flexible human researcher might findgiven enough time and would consider relevant. For example, keyword andrelationship based searching may return videos or other media related toa user query as an alternative to returning webpage results. However,the media results are based on a relationship to the query term itselfand not any other search results. Thus, media information related asearch result that may be useful to a researcher in identifying andunderstanding both the content and context of a search result is notavailable using these search techniques.

SUMMARY OF THE INVENTION

Certain embodiments herein provide for a system and computer-implementedmethod for generating a video search engine results page (VSERP). Itshould be understood that while the term “video search engine resultspage” is used herein to describe a result page from a search, it is notlimited to providing video-based results to a user. Other results thatinclude audio, photographs and other media objects are alsocontemplated. Certain embodiments also provide for retrieving mediaobjects related to a search result and displaying the search result andmedia objects on a VSERP. Certain embodiments generate summaries ofsearch result objects and extract high-value terms and phrases in orderto match media objects to the search results.

In one embodiment, a system to generate a VSERP is provided. First,search results are retrieved based on a query from a user. Well knowncompanies that provide such search results include GOOGLE, YAHOO andMICROSOFT. Once those search results have been returned, the systemanalyzes the content on each web page that was returned in the search,thus generating a content analysis. The content from that web page (orthe resultant content analysis) is then used as the basis to retrievemedia objects, such as movies, videos, audio and images that relate tothe content on each returned web page. Media objects are obtained, forexample from an internet video storage website such as YOUTUBE. Based onthe content of each search result and information surrounding the mediaobjects, related media objects are matched to the web page that wasfound as a result of the original search. The search results arereturned to the user along with the matched media objects. Thus, theVSERP contains not only search results but also media directly relatedto the each search result. It should be realized that this systempreferably does not directly use the user's query to return mediaobjects, but rather takes the content that was found during the searchas the basis for additional searches to find media objects that relateto the returned content.

In one embodiment, a system to generate a summary of a document in anelectronic database is provided that may be utilized in the process ofgenerating a VSERP. According to this embodiment, vectors areconstructed from the content that is returned during a search. Thesevectors contain data representing certain relationships between objectsin the returned content. The content may be divided into sections, andsections may be scored using the data contained in the relationshipvectors. Those sections receiving high scores are utilized to create anobject summary. In one embodiment, an object summary is created for eachresult in search engine results page. High-value terms and phrases maythen be extracted from each summary. This extracted information from thecontent that was returned from the user's search can be used to findmedia objects that match the returned content and thereby create a VSERPwhich displays returned web pages and the media associated with thoseweb pages.

In another embodiment of the invention, the high value terms or sectionsof the object extracted from the summary can be used as tags to providecontextually related search terms. In other embodiments, the objectsummary acts as an executive summary of a document allowing a user toquickly comprehend the content and context of the document.

In one embodiment, there is a method of searching for media objectsassociated with a search term, comprising receiving a first search termfrom a user, performing a first search of Internet data, wherein saidfirst search identifies web pages related to said first search term,analyzing said web pages to determine additional search terms relatingto said first search term, and performing a second search for mediaobjects relating to said additional search terms.

In one embodiment, there is a method of retrieving media objectsassociated with a query, comprising receiving a query from a user,retrieving, based upon the query, at least one search result, generatinga content analysis for the at least one search result, and retrieving,based upon the content analysis, at least one media object.

In another embodiment, there is a system for searching for media objectsassociated with a first search term, comprising a search moduleconfigured to retrieve, based upon a received first search term, atleast one web page, an analysis module configured to analyze said webpage to determine additional search terms based on said received query,and a media search module configured to retrieve, based upon theadditional search terms, at least one media object.

In another embodiment, there is a system for searching for media objectsassociated with a search term, comprising means for receiving a firstsearch term from a user, means for performing a first search of Internetdata, wherein said first search identifies web pages related to saidfirst search term, means for analyzing said web pages to determineadditional search terms relating to said first search term, and meansfor performing a second search for media objects relating to saidadditional search terms.

In yet another embodiment, there is one or more processor-readablestorage devices having processor-readable code embodied therein, theprocessor-readable code for programming one or more processors within acomputer to perform a method of searching for media objects associatedwith a search term, the method comprising receiving a first search termfrom a user, performing a first search of Internet data, wherein saidfirst search identifies web pages related to said first search term,analyzing said web pages to determine additional search terms relatingto said first search term, and performing a second search for mediaobjects relating to said additional search terms.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a representation of one embodiment of a video search engineresults page.

FIG. 2 is a flow chart for one embodiment of a system for generating arelationship network.

FIG. 3 is a flow chart for one embodiment of a system for generatingvectors for use with a relationship network based on an electronicinformation database containing text documents.

FIG. 4A shows a sample document from an information database containingtext documents.

FIG. 4B shows the document of FIG. 3A after it has been parsed.

FIG. 4C shows one embodiment of a frame for use with the sample data ofFIGS. 3A and 3B.

FIG. 5 shows a sample associative memory module for the term “red” fromFIG. 4 at a state where the current term being analyzed in the frame isthe core term “red.”

FIG. 6A shows the associative memory module for the term “red” after thesystem completes its analysis of the information database containing thedocument of FIG. 3A.

FIG. 6B shows the sample query object vector for the associative memorymodule of FIG. 6A.

FIG. 7 shows a sample flow chart for a network generation engine.

FIG. 8A shows a sample exclusion filter vector applied to a query objectvector.

FIG. 8B shows one sample method to generate an expanded query objectvector using the filtered query object vector of FIG. 8A.

FIG. 8C shows one sample method to generate expanded associated objectvectors using the filtered query object vector of FIG. 8A.

FIG. 8D shows one sample method to use expanded associated objectvectors with an expanded query object vector to find associated termsbetween the associated object vectors and the expanded query objectvector in order to produce search results for a query.

FIG. 9 shows a graph visualization for a relationship network created inresponse to a query for the term “red.”

FIG. 10 illustrates a relationship network system according to oneembodiment.

FIG. 11 is a flow chart for one embodiment of a system for displaying avideo search engine results page.

FIG. 12 is a flow chart for one embodiment of a system for generating avideo search engine results page.

FIG. 13 is a flow chart for one embodiment of a system for summarizingobjects in an electronic database.

FIG. 14 is a flow chart for one embodiment of a system for scoringsections of an object in an electronic database.

FIG. 15 shows one embodiment of a system for analyzing for scoringsections of an object in an electronic database at a state where thefirst section and first term have been selected.

FIG. 16 shows a sample query object vector.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description, reference is made to the accompanyingdrawings which show, by way of illustration, specific embodiments andapplications of the invention. Where possible, the same referencenumbers are used throughout the drawings to refer to the same or likecomponents. In some instances, numerous specific details are set forthin order to provide a thorough understanding of the present disclosure.The present invention, however, may be practiced without the specificdetails or with certain alternative equivalent components and methods tothose described herein. In other instances, well-known components andmethods have not been described in detail so as not to unnecessarilyobscure aspects of the present disclosure.

One embodiment of the invention is a computer method and system thatcreates and displays a VSERP. In one embodiment, a list of searchresults generated in response to a user query, which may take the formof a search term, is returned from one or more search engines such asGOOGLE, YAHOO or the like. Media objects are retrieved and matched toeach of the search results, and the media objects or links to theobjects are displayed along with the related search results to a user.

FIG. 1 is a representation of a VSERP 100 according to one embodiment ofthe current invention. VSERP 100 comprises a search entry box 110 andlist of search results 120, 130, and 140. Search results 120, 130, and140 were generated in response to a user query term 101. In the exampleshown, query term 101 is “tree.” Results 120, 130, and 140 are thus allrelated to the query term 101, though each result has different content.For example, search result 120 is a webpage containing information aboutdifferent species of trees, search result 130 is a webpage containinginformation about seasonal color changes in tree leaves, and searchresult 140 is a webpage describing family trees and genealogy.

Each result in VSERP 100 is associated with two media objects. The mediaobjects are extracted from electronic databases that were found to havemedia content matching each of the individual search results. Forexample, the electronic database may be an internet video storagewebsite such as YOUTUBE, GOOGLE VIDEO, YAHOO! VIDEO, or the like.Extracted media objects are matched to the search results, and the mediaobjects most related to each search result are returned. In theembodiment shown, media objects 121 and 122 are the results found to bemost relevant to search result 120. Media object 121 is a time-lapsevideo showing the flowers of a cherry blossom tree blooming. Mediaobject 122 is an image of an oak tree leaf. Media objects 121 and 122were both extracted from an electronic database and matched to searchresult 120. Media objects 131 and 132 were matched to search result 130,and are therefore related to that webpage. Media object 131 is an imageshowing a forest of trees changing color in the autumn and media object132 is a video of tree leaves changing color and falling off a tree.Search result 140 is associated with media objects 141 and 142. Mediaobject 141 is an image showing a template of a family tree, with spacesto write in a person's name, their father, their mother, siblings, andother relatives. Media object 142 is an image of a completed family treefor an existing family. As can be seen, each media object returned isranked based upon its relation to a search result, and not to theinitial query term 101. Thus, a VSERP preferably contains media objectscontextually related to each search result that may be more useful andprovide more relevant media than would otherwise be provided. Theembodiment shown contains three search results and two media objectsrelated to each result. Other embodiments may contain any number ofsearch results and any number of related media objects. In a preferredembodiment, ten search results are displayed on a VSERP at one time, andthree media objects are related to each search result.

Overview of System for Generating Relationships

One embodiment of the invention is a computer method and system thatcreates and discerns relationships between different items in acollection. In one embodiment, a many-to-many relationship is createdbetween data items in a data set. As one example, the data items may begenes, and the data set may be the GENBANK gene database. As will bedescribed in more detail below, embodiments of the system analyze thedata items in the data set and thereafter create variable length datavectors, such as query object vectors, that reflect the relationshipsbetween the data items in the dataset. The data vectors can then bestored and used as part of data mining tool which analyzes relationshipsbetween the data items. For example, one may search for all genes inGenbank that relate to stomach cancer.

In one embodiment of the invention, the data vectors that markassociations between data items are created by first analyzing directcorrelations between two data items, and then looking for further,hidden, associations between the data items. In one embodiment, thesehidden relationships are determined by iteratively analyzing thedistance that each term in the dataset has from other terms. Thus, forexample, the more times that two words are found to be associated withone another in the data set, the closer the relationship between them isformed. In certain embodiments, terms are analyzed by moving a “frame”through each data item. For example, if the data item is a document, theframe may move through the document one line at a time, but coveringthree lines. As the frame moves down each line of the document, thedistance between terms within the frame is analyzed. During thisanalysis, data vectors are created which store the relationships betweeneach term in the frame. In one embodiment, each term within the entiredataset is represented by one vector. That vector provides the distancesand relationships between that term and its related terms.

One embodiment of the invention is a system using the stored datavectors to create summaries of documents, objects, or collections. Inthese embodiments sections of objects are scored based on theinformation in the data vectors. The section scores are compared andthose sections having the highest scores, representing the most relevantinformation, may be returned as a summary to a user. For example, themost relevant sentences or paragraphs of a document may be returned as asummary. Related embodiments may extract a group of highly relevant tagsfrom a summary. These tags are matched to media objects in order tocreate a VSERP. In other embodiments, the tags may be returned assuggested search terms.

Another embodiment of the invention is a system and method of using thestored data vectors to provide useful results of a search inquiry. Whena person or machine inputs a term as part of a search, the data vectorfor that term is located, and the terms most relevant to the search termare identified from the data vector. The system then retrieves the datavectors for the most relevant terms in order to expand the search. Theterms that are related to the most relevant terms can then beidentified, and the process can continue to build a relationship networkbetween the original search term, and all of its related terms. Once thequeries are executed and the vectors containing the most relevant termsare scored, a relationship network is built. The resulting network ofthe submitted term may then be prepared for visualization for furtherinterpretation. In one embodiment, the terms are displayed on a computerscreen with a web of links showing how related each search term was toits results. To ensure that the submitted terms stay within a specificcontext when a relationship network is being built, a thematic contextin the form of a filter can be used to control the kind of relationshipsextracted within the resulting network.

The systems and methods disclosed herein allow a user to interactivelyengage in information mining, hidden association and connectionextraction, relationship network construction and comparison of objectswhile interactively applying thematic context controls to refine thetype of relationships extracted. The systems and methods provide theuser with information on how objects within the information databaserelate to one another, in what contexts they are related, and thestrength of their relationship.

By combining an interactive role for the user, similar to what aresearcher engages in during the process of experimentation, andapplying it to an iterative process of automated text mining methods,certain embodiments discussed herein give the user the ability to choosethe direction and define relationships as connections are made betweenobjects of interest in the information searched. Interactively definingand extracting relationships between objects, themes and other contextsprovides a valuable level of precision for relationship exploration anddiscovery in text.

For example, if a user was searching for Baroque compositions in anelectronic information database such as the Internet, the user maysubmit the term “Baroque” to the relationship network system. The usermay also choose to direct the search in the direction of Baroque musicby using a filter term such as “compositions” in order to avoid resultsrelating to Baroque art. The system would then not only provideinformation on compositions strongly associated with the term “Baroque,”but also for compositions strongly associated with terms related to“Baroque,” such as composer names “Bach” and “Handel,” compositionsinvolving instruments associated with Baroque music, such as “viola dagamba” or “harpsichord”, or the related art period, “Classical,” and soon.

In one embodiment, the relationship network system disclosed herein maybe used for term disambiguation, which provides the ability todistinguish two strings of characters that are exactly the same but thathave different meanings dependent upon context such as acronyms thatdouble as identifiers or symbols or actual words. For example, the word“cleave” has two definitions that are opposite of one another.

FIG. 2 shows a process 200 for generating a relationship network usingan electronic information database. In certain embodiments, anelectronic information database may include, but is not limited to, acollection of characters or other forms of text, images, audio, video,or any other data that may be analyzed electronically. Objects or termswithin the information database may thus be documents, characters,words, images, songs, or videos (“terms”).

In the embodiment illustrated, the system first selects an electronicinformation database to process at a state 201. In one example, thedatabase is a database of musical compositions. The system then createsvectors for terms within the database at a state 202. The vectors arecreated in a way to capture the different strengths of relationshipsbetween compositions within the database. Once the vectors are created,the system receives a query “Q” from the user at a state 203. A query isundertaken, for example, when a user would like to find compositionssimilar to composition listed in the query Q. In certain embodiments,the system may create the vectors before receiving a query in order toreduce data processing expenditures in response to the query. In otherembodiments, the vectors may be created after the query is received.Although in certain embodiments a vector is used to store relationshipsbetween terms, other data structures may be used in other embodiments.In certain embodiments using vectors, the vector space representationscheme uses variable length query object vectors. The variable lengthvector may have a plurality of component values or elements that aredetermined based on relationships between terms. In addition, thevariable length vectors may be sized based on the number of associatedterms within each vector.

In certain embodiments, associated terms are terms that have either adirect or indirect relationship with each other. In some embodiments,the one term is a “first” term and the second term is a “core term”. Incertain embodiments, a direct relationship is where a core term is foundwithin the same frame in a vector as the associated term. In certainembodiments, an indirect relationship is where a core term and theassociated term each share a common term in their respective vectors.Other relationships between terms may also be generated for use withcertain embodiments discussed herein.

Returning to FIG. 2, in response to a query for term Q from a user atthe state 203, the system then generates a relationship network for Q ata state 204 based on the variable length vector(s) for the term Q. Incertain embodiments, a relationship network is comprised of a network ofrelationship vectors whose connections to each other, and the strengthof those connections, are based on shared unique attributes within adefined context and theme. Contexts and themes are discussed morespecifically below. Once the relationship network has been generated atthe state 204, the system may then return terms that are associated withQ at a state 205. For example, the returned terms may point tocompositions that are by the same composer as Q, compositions related toQ, or recommendations based on Q.

1. Generating Vectors for a Relationship Network

FIG. 3 is a flow chart for one embodiment of the process 202 ofgenerating variable length vectors from data stored within a database.The process 202 gathers each document in the database at a state 301.For each document that is gathered, the document is parsed at a state302 in order to remove irrelevant or low value data, such as stop-words(common words such as a, of, as, the, on, etc.). After each document hasbeen parsed at the state 302, the information database contains onlyvaluable terms.

Then, for each parsed document, the system inserts a frame at a state303 in the document. The frame can be thought of as an overlay thatcovers one or more lines of text in the documents. For example, theframe may cover three lines or sentences in the document. Once the framehas been inserted at the state 303, the process 202 moves to a state 304wherein the first term in the first line processed in the frame isselected. FIG. 4C shows one embodiment of a frame 440 for use with thesample data illustrated in FIGS. 4A and 4B. After the first term in theactive sentence of the frame is selected at the state 304, a set ofrelationship data is generated between the first term (“core term”) andthe other terms within the frame (“associated terms”) at a state 305.The system records the relationship data for the core term, whichincludes data such as a calculated distance score for each core termfrom the first term. In certain embodiments, the relationship data maybe stored in an associative memory module, as shown in FIG. 5. Once therelationship data has been generated for the first term, the process 202moves to a decision state 306 wherein a determination is made whetherthe last term in the active sentence of the frame is being analyzed. Ifthe current term is not the last term, then the process 202 moves to astate 307 wherein the next term within the frame is captured. Theprocess 202 then returns to the state 305 to calculate the relationshipdata between the newly captured term and the other core terms within theframe at the state 305. If the term being processed is the last term inthe active sentence of the frame, then the process 202 moves to a state308 wherein the frame is moved ahead by one sentence or line in thedocument under analysis. If the term is not the last term in the activesentence for the frame, the process 202 moves back to state 305.

Once the process 202 has moved the frame ahead by another line orsentence, a determination is made whether or not the frame is at the endof the document at a decision state 309. If a determination is made thatthe process 202 is not at the end of the document, then the process 202returns to the state 304 wherein the first term within the activesentence of the moved frame is selected. If a determination is made thatthe frame is at the end of the document, then the process 202 moves to adecision state 310 where a determination is made whether or not theprocess is at the last document in the database. If the process 202 isnot at the last document in the database, then the process 202 moves toa state 311 wherein the next document within the database is selected.The process 202 then returns to the state 303 wherein a frame isinserted into the newly gathered document.

If a determination is made at the decision state 310 that the process202 is at the last document, then the process moves to state 312 whereit retrieves the recorded relationship data, such as from theassociative memory module, for the first term in the database. Then theprocess moves to state 313 where a variable length query object vectoris created using the relationship data from state 312. In certainembodiments, the relationship data values from state 312, which may bestored in a query object vector, may be enhanced when stored in thequery object vector. Examples of enhancing the relationship data valuesinclude increasing the data values of unique associations and decreasingthe data values for common associations. FIG. 6B shows the sample queryobject vector for the associative memory module of FIG. 6A. Next, theprocess moves to decision state 314 then checks to determine if the termanalyzed is the last term in the database. If it is not the last termanalyzed, the process moves to state 315 wherein the next term withinthe database is selected. The process 202 then returns to the state 313wherein a query object vector for the next term is created. If adetermination is made at the decision state 314 that the process 202 isat the last term, then the process terminates at the end state 316.

FIG. 4A shows a sample document 400 from an information databasecontaining text documents. FIG. 4B shows the stored data from thedocument of FIG. 4A after it has been parsed 410. As it can be seen fromthe differences between FIGS. 4A and 4B, in this embodiment the systemremoved stop-words such as “they” 401 “from” 402 “until” 403 and“they're” 404 and also organized each sentence according to theidentification of the document 411 it was found in and its terms 412.

As shown in FIG. 4C, one embodiment of the context or frame 440 consistsof associated terms surrounding and ultimately associated with thecurrent, core term being analyzed in the frame, “red” 450. In oneembodiment, the frame 440 and the space it encompasses are constructedby using distance thresholds within documents. For example, in FIG. 4C,the distance threshold is one sentence before and one sentence after thesentence containing the core term being analyzed 450. If a term iswithin the distance threshold, it is considered an associated term andit becomes part of the context frame 440. On the other hand, if a termis outside the distance threshold, it will not become part of thecontext frame 440 and does not receive a distance score (also referredto as a score association) to the core term. Using the number of wordsin a document as well as number of sentences, paragraphs, characters orother objects, distance thresholds can be calculated and the size of theframed context 440 will grow and fluctuate as documents are read in andnew statistical data is gathered. In one embodiment, wherein the digitalcontent to be analyzed is raw text documents, the frame 440 is set tothree, four or five sentences per frame. The example in FIG. 4C has athree sentence context frame 440.

The system may move the frame 440 through the documents or other parseddata which comprise the information database. As the frame is moved lineby line through a set of documents, terms can be automaticallyassociated with one another including an identifier representing theoperative document 411. As terms flow in and out of the frame that movesthrough the documents, associated terms can define their strength ofassociation to the core term by distance scores. For example, in FIG.4C, after the system has calculated the distance scores for the coreterm “red,” the focus of the frame will move to the next term, “pink,”until the focus reaches the final term in the middle line of the frame,“raspberry.” After the system has calculated the distance scores forterms associated with the term “raspberry”, the frame will advance byone line and the core term focus will begin with the first term on thenext line, “Hummingbirds.” Furthermore, the sentence beginning with theterm “bloom” will flow out of the frame and the sentence beginning withthe term “one” will flow into the frame.

By giving a distance score to each associated term, each core term 450in the document becomes a statistically important object containing afamily of relationship scored associative terms as elements of itsassociative memory module. The distance score between two terms may thenbe used to create a relationship score between two terms after theprocess completes analysis of the entire information database. Forexample, in one embodiment, distance scores between two terms as theyappear repeatedly within a frame throughout the information database maybe summed to create a relationship score.

Frame 440 usage in single documents becomes especially advantageous whenrelationship scores are generated over thousands or millions ofdocuments. In certain embodiments herein, significant relationshipsbetween words are defined over time by strong and unique connectionsbetween two or more terms. Relationship scores to a term can be comparedto the way a person might learn by repetition. A person will tend toremember and associate two terms together if he hears them together on arepeated basis, whereas a person may not remember or associate two termstogether if he does not hear them together very often. In certainembodiments discussed herein, the system gives a high relationship scoreto two or more terms which appear often together. In certain otherembodiments, two or more terms sharing a very unique set of attributesare scored highly.

As discussed above, the system may store relationships between a coreterm 410 and its associated term in file called an associative memorymodule that is created for the core term. In one embodiment, anassociative memory module is a database schema storing informationrelated to statistical and distance-based object associations, as wellas document statistics. The associative memory module may thusadvantageously capture meaning sensitivity in the data to be searched,which requires that the closeness of every pair of terms be known,scored for distance and stored. Thus, associative memory modules mayadvantageously store information such as words, paragraphs, searchqueries, objects, documents, document identifiers, parts of images,parts of terms, parts of text, parts of sequences or any piece of anobject that has been split into parts, terms and documents, and manyother types of information items similarly represented, such asnumerical, financial, and scientific data. In one embodiment, everyassociated term in an associative memory module and vector is also thecore term of its own associative memory module and vector, therebyenabling a high dimension many-to-many scored associative relationshipnetwork. In certain embodiments, this in turn enables strong comparisonto occur between, for example, parts of terms, between terms, and termsand the documents they appear in.

In certain embodiments, the length of associative memory modules andvectors may be limited in order to facilitate faster creation of therelationship network or due memory storage constraints since the lengthof the vector or module may affect the size of the database and thesystem's performance capabilities. In other embodiments, an associativememory module or vector may contain as many elements as may besupported. In certain embodiments, the system may present a certainnumber of terms with a high score, or terms with a score above a certainthreshold value in order to best represent the information databasequeried and to facilitate viewing by a user.

FIG. 5 shows a sample associative memory module for the term “red” 500from FIG. 4C at a state where the current term being analyzed in theframe 400 is the core term “red” 450. The associative memory module 500shown has three sections: statistics related to the term 510, statisticsrelated to documents containing the term 520, and statistics related toassociated terms 530. In the embodiment displayed, the first section,statistics related to the term 510, may contain information such as thenumber of occurrences of the term in the text analyzed 511, the numberof sentences that contain the term 512, the number of other termsassociated with the core term 513, and the number of associationsbetween other terms with the core term 514. Since the associative memorymodule 500 displayed only contains data through analysis of the term“red” 450 in the first document analyzed in the database (FIG. 4A), thedata in FIG. 5 reflects the incomplete analysis. Thus, since the term“red” 450 has occurred only once so far, and in only one sentence 452,the number of occurrences 510 and number of sentences 511 for the term“red” 450 both equal one. Similarly, since all eighteen of the termsanalyzed so far are also all of the terms currently in the frame 440,they are all associated 513 with the term “red” 450. Furthermore, sincenone of these associated terms have yet appeared twice, they are alleighteen individual associations 514 for the term “red” 450.

The document statistics section 520 advantageously identifies documents521 that contain the term, the number of sentences in the document thatcontain the term 522, and a score for the document in relation to theterm 523. In the sample shown, only one document 524 is listed becauseit is the only document analyzed that contains the term “red”. Thedocument 524 is identified by its title, although any other well knownidentification system may be used to record document identifications,such as a uniform resource locator (“URL”) address. Furthermore, onlyone sentence 525 that contains the term “red” has been found in thedocument. Consequently, a score 526 of one has been assigned to thatdocument. In the embodiment shown, the score 526 associated with adocument is the number of appearances of the term within the document,although in other embodiments other scoring methods may be used.

The associated terms section 530 includes, but is not limited to, datasuch associated terms 531, the number of occurrences of each associatedterm in relation to the core term 532 and the corresponding distancescore for the associated term/core term pair 533. In other embodiments,the associated terms section 530 may also include data on the number ofsentences processed so far that contain the associated term in relationto the core term and the distance of the associated term to the coreterm.

Distance scores 533 to measure associations between terms are appliedwithin the moving frame. For example, FIG. 4C shows a three sentenceframe 440 surrounding the core term, “red”. As the frame 440 and itscore term focus 450 moves through the document a calculation is appliedto assign distance scores to each term within the frame 400 in relationto the core term 450.

A distance score 533 may be calculated by any number of well knownmethods. Furthermore, in order to give greater value to associated termsin closer proximity to a core term, the distance score values 533assigned to associated terms as their distance to the core termincreases may advantageously be decayed. This may advantageously beapplied using the Fibonacci sequence in reverse. In other words, in oneembodiment using the Fibonacci sequence in reverse, the distance scorefrom the core term to an associated term is:S_(ij)=φ^(Δx),

where:

-   -   S_(ij)=distance score between core term i and associated term j,    -   φ=0.618 is the Golden Ratio component “phi”⁵⁵⁴ , and    -   Δx=|x_(i)−x_(j)| is the relative position between core term i        and associated term j.        -   ^(†)φ is the decimal component of the Golden Ratio            φ=1.618034.

Returning to FIG. 5, the distance score 536 using this equation for theassociated term “cardinal” to the term “red,” which are neighboringterms (Δx=1), is 0.618=0.618¹. Similarly, the distance score 537 for theassociated term “bloom” to the term “red” is 0.008=0.618¹⁰, since“bloom” is ten terms away from “red” (Δx=10). In certain embodiments, asthe system encounters a second occurrence between an associated term anda core term separate from the first occurrence, then the system may addthe distance score of the second occurrence to the first occurrence inorder to keep a running total of the distance score for the association.For example, in FIG. 5, if the system encounters the term “cardinal” 534again within a frame containing “red”, and the distance score for thesecond occurrence is 0.008, then the system may update the distancescore 536 for “cardinal” in the “red” associative memory module 500 tobe 0.626=0.618+0.008. In other embodiments, other methods may be used toupdate a distance score value as the system processes an informationdatabase.

Calculations based on Fibonacci's number may be advantageously usedbecause sequences based on the ratio of successive Fibonacci numbers,the Golden Ratio, are found in many natural phenomena, including biologyand materials science. Fibonacci's number may thus have a relationshipto grammar and human generated patterns and an effect on theinterpretation of information.

In another embodiment, the Enhanced Exponentially Weighted MovingAverage (EEMA), a variation of the EWMA (Exponentially Weighted MovingAverage) time series calculation, may be used to compute distance scoresbetween terms within a frame. A sample equation using the EEMA may bedefined as:EEMA=1/((K*(C−P)+P)

Where:

-   -   C=Position of the core term    -   P=Previous period's Simple Moving Average (SMA)    -   N=Number of periods for EEMA    -   K=e^((−C/5.0)) Smoothing constant

In yet another embodiment, a standard exponential decay algorithm can beapplied. Below are two equations for exponential decay that can be usedto calculate distance scores:

If core term i comes before associated term j, then Sij = 1/e(j − i) Ifcore term i comes after associated term j, then Sij = 1/e(i − j) whereSij = relationship score between object i and j,

FIG. 6A shows the associative memory module 600 for the term “red” afterthe system completes analysis of the information database containing thedocument of FIG. 4A. In the sample associative memory module 600, thesystem has determined that the information database analyzed containstwelve occurrences 611 of the term “red” in a total of twelve sentences612. Furthermore, there are 319 terms associated with “red” and 450associations between those terms and “red”. Whereas the document“Gardening Journal” 625 contained four sentences 626 totaling fouroccurrences of “red”, the document “Top News Stories” 628 only containedone sentence with one occurrence 630. Additionally, while the associatedterm “cardinal” 634 had six associations with red for whose individualdistance scores summed to equal a total distance score 636 of 4.124, theassociated term “paste” 637 only had one associated occurrence with“red” for a total distance score of 0.008.

After the system processes each document in the information database,each associative memory module may be used to create a query objectvector. FIG. 6B shows a sample query object vector 650 created from theassociative memory module 600 of FIG. 6A. In the embodiment shown, thedistance score 633 from the associative memory module 650 is used tocalculate the relationship score 653 for the query object vector 650 byemphasizing common associations, as will be discussed in further detailbelow. The system then ranks the associated terms in the query objectvector 650 according to their relationship scores 653. For example, inFIG. 6B, the associated term “Cardinal” 654 is ranked first because ithas the highest relationship score and the term “Paste” 655 is ranked at319, which equals the total number of terms associated with “red,”because it has the lowest relationship score. Each associative memorymodule is thus used to create a query object vector 213.

FIG. 6B thus illustrates one advantage of the systems and methodsdescribed herein. In keyword based searches, if a user looking for redsweaters used the term “red” in her query, then she would only receiveresults where the sweaters were specifically listed with the term “red.”On the other hand, if the user submitted the search to an embodiment ofthe system described herein, the user would not only receive results for“red” sweaters, but for sweaters with other shades of red, such ascardinal, maroon and raspberry.

In certain embodiments, the system may advantageously use data from anassociative memory module in order to create a different relationshipscore values for a query object vector. For example, in one embodiment,the distance score may be modified with the aim of emphasizing uniqueassociations, such as to help in finding hidden relationships. Hiddenrelationships may be used to assist in hypothesis formulations bypresenting a list of possibly important new relationships unknown to theuser. In one embodiment, the following uniqueness function may be usedto calculate a relationship score emphasizing uniqueness:U _(ij) =S _(ij) ·B _(ij)

where:

-   -   S_(ij)=Distance-based relationship score between term i and j    -   B_(ij)=Bias for term i of association with term j,        -   where:            B _(ij) =A _(i) /A _(j)            -   A_(i)=Total number of associations of term i            -   A_(j)=Total number of associations of term j

In another embodiment, the distance score may be modified with the aimof emphasizing common associations such as to generate a cleardefinition based on direct associations. Direct associations can be usedto generate a list of very similar objects. In one embodiment, thefollowing commonality function may be used to calculate a relationshipscore emphasizing commonly associated terms:B _(ij) =A _(j) /A _(i)

where:

-   -   A_(i)=Total number of associations of term i    -   A_(j)=Total number of associations of term j

Thus, by the time the process of FIG. 3 completes, each term in eachparsed document will have its own query object vector; i.e., each termwill be a core term for a query object vector and an associated term forother term's query object vectors. In certain embodiments, each queryobject vector may either emphasize unique or common relationships.Furthermore, in certain embodiments, each document will also have itsown associate memory module and query object vector. These vectors maythen be used to build a relationship network.

2. Building A Relationship Network

FIG. 7 shows a process 700 for a network generation engine for use withembodiments of the relationship network discussed above. Specifically,disclosed is one embodiment for generating a relationship network usingthe query object vectors generated from an electronic informationdatabase containing text documents as described above. In response to asearch query term inputted by a user, a relationship network may begenerated from the extraction of relationships from query object vectorsbased upon the search query term. In certain embodiments, therelationship network would be comprised of a network of expanded vectorsof terms, their connections to each other and the strength of theseconnections, where the connections are based on shared attributes withina defined frame. Although the sample flow chart illustrated discusses anembodiment using text documents and terms, in other embodiments, thequery term may be audio data, video data, image data, or any other kindof electronic data.

First, a user submits at least one query term, Q, to the system at astate 701. In certain embodiments, multiple terms may be submitted tothe system, and may be treated as one query term or a multiple of queryterms. In certain embodiments, if Q does not exist in the informationdatabase, then the system does not return any data. In response toreceiving the query, the system retrieves the vector for the query term,the query object vector (“QOV”) at a state 702. The process 700 thenmoves to a state 703 wherein the user or system configures a filter foruse with the query in order to focus the query results. This filter maybe set, by for example, filtering terms out of the vector retrieved forthe search term Q at the state 703. This will be discussed in furtherdetail below with reference to FIG. 8A. Next, the system expands thevector into an expanded QOV at a state 704. This process will bediscussed in further detail below with reference to FIG. 8B. The process700 then moves to a state 705 wherein the system uses the QOV togenerate expanded associated object vectors (“AOV”). This will bediscussed in further detail below with reference to FIG. 8C. The systemthen moves to a state 706 to find associated terms between the expandedAOVs and the expanded QOV. Search results for the query Q are thenprovided at a state 707. The process of providing search results will bediscussed below with reference to FIG. 8D. Finally, the process 700presents a visual representation of the relationship network based onthe query results.

In one embodiment, the system uses filters, such as forms of ontology ofrelated themes and categories, to control the kind of relationshipsderived during the search process and to ensure that terms stay within acertain defined context when the relationship network is being built. Incertain embodiments, filters may be employed because the terms selectedfor the filter also exist in the information database being searched, sothe filter terms thus have vectors of their own. The filter may besupplied along with the query in order to focus the query results. Thefilter can be a list of words, symbols or objects by which the resultsof a query are controlled. For example, the filter phrase “genes andinferred relationships to drugs” may be used for a genomic search doneon an information database related to genetic data.

In certain embodiments, the filter may be a complete vector wherein itselements represent the entire set of frame data or context in a databaseof documents to control the relationship extraction process. Any searchresults that are found to intersect with the vector-filter will beprocessed according to the type of filter used.

Many different kinds of filters may be enlisted for use with the systemsand methods disclosed herein. One type of filter, an exclusion filter,can actively remove terms and vectors which do not match the filter.Exclusion filters may be used to assure that elements from a specifictheme are removed from the query object vectors and associated objectvectors for any aspect of the process. FIG. 8A shows a sample exclusionfilter vector 810 containing the terms Z₁ to Z_(n). The filter vector isapplied to the query object vector 820 retrieved for query Q 801 inorder to focus the results of the query. As shown in FIG. 8A, the systemadvantageously removes instances of terms that appear in the filtervector. The terms Z₁, Z₂, and Z₃ have been filtered from the final queryobject vector 825 because those terms appear in the exclusion filter810.

On the other hand, a selection filter can actively select terms andvectors which match the filter. Selection filters may be used to assurethat only elements from a specific theme are used for a specificprocess. In one embodiment, the process includes the selection of topquery term vector elements and associated term vector elements forgeneration of expanded query term vectors and associated term vectors.Filter elements also effect the selection of final terms being used inthe expanded query term vector to expanded associated comparison andassociation score calculation.

Another type of filter, a weighting filter, may adjust the relationshipscores of certain terms and vectors in order cause the terms or vectorsto be reordered. Weighing filters may be used to alter the weight of aspecific group of terms, thereby affecting their impact on the algorithmprocess and calculation results.

Filters may advantageously be applied during any point wherein thesystem is expanding the query object vector retrieved in response to aquery. The use of filters results in the ability of the system to baserelationships on specific sets of terms which may comprise a theme.Without theme filtering, the system might retrieve inferredrelationships of all kinds which may not be beneficial if it is notknown what kind of relationships to look for. For example, a usersubmitting the search query term “red” to an information databasewithout a filter might receive very broad results. On the other hand, ifthe user employs a selection filter, which would exclude all terms notfound in the filter, such as the filter phrase or vector “flowers” as acontext for “red,” specific terms relating to red colored flora willmost likely be found in the query results. In certain embodiments,filters may be predefined and interchangeable in order to allow a userto tailor a search query. Creating a network of term relationships withthis kind of context control allows for previously unidentifiedconnections to be brought to the fore as a user of the system mightdesire to find what relationships to this query term exist in aspecified context.

FIG. 8B is a data flow diagram that shows one exemplary method ofgenerating an expanded QOV 850 using the filtered QOV 825 of FIG. 8A.First, the system identifies the thirty strongest terms, A₁ to A₃₀ 826,related to the query term Q 801. These thirty strongest terms are addedto the beginning 826 of the expanded QOV 850. Next, the system retrievesthe vectors for each of those thirty terms, A₁ to A₃₀ 830, and insertsthe top three strongest terms in each of those thirty vectors 831 (i.e.,A_(1,1) to A_(1,3) for A₁, A_(2,1) to A_(2,3) for A₂, . . . A_(10,1) toA_(10,3) for A₁₀) to complete the expanded QOV 850. Although theembodiment of the system shown selects thirty terms for processing, inother embodiments, any other number of terms may be used for processing.

FIG. 8C is a data flow diagram showing one method of generating anexpanded AOV 875 using the filtered QOV 825 of FIG. 8A. First, thesystem identifies the thirty strongest terms, A₁ to A₃₀ 826, related toQ 801, retrieves their vectors 827, and begins an expanded AOV 875 foreach term A₁ to A₃₀. Then the system identifies the three strongestterms from the first dimension vectors related to each of A₁ to A₃₀,(i.e., A_(1,1) to A_(1,3) for A₁, A_(2,1) to A_(2,3) for A₂, . . .A_(30,1) to A_(30,3) for A₃₀) 830, adds to the corresponding expandedAOV 875, A₁ to A₃₀, and retrieves their vectors 831. Similarly, thesystem retrieves the three strongest terms from the second dimensionvectors related to each A_(1,1) to A_(30,3), (i.e., A_(1,1,1) toA_(1,1,3) for A_(1,1), A_(1,2,1) to A_(1,2,3) for A_(1,2), . . .A_(30,30,1) to A_(30,3,3) for A_(30,3)) 840 and retrieves their vectors841. Once more, the system retrieves the three strongest terms from thethird dimension related to each A_(1,1,1) to A_(30,3,3) (i.e.,A_(1,1,1,1) to A_(1,1,1,3) for A_(1,1), A_(1,1,2,1) to A_(1,1,2,3) forA_(1,2), . . . A_(30,3,3,1) to A_(30,3,3,3) for A_(30,3,3)) 850. The topthree associated terms from the third dimension vectors 850 are theninserted after the first dimension terms 830 already in the expanded AOV875 to complete the expanded AOV 875. Although FIG. 8C shows thegeneration of an expanded AOV 875 for A₁, in the embodiment shown theprocess produces a total of 30 expanded AOVs for each A₁ to A₃₀ 826.

FIG. 8D is a data flow diagram that shows one exemplary method of usingexpanded AOVs 875 with an expanded QOV 850 to find associated termsbetween the AOVs 875 and the expanded QOV 850 in order to produce searchresults for the query Q 801. The expanded vectors 850 and 875 are passedto a function that determines similarity between intersecting terms inthe expanded vectors 850 and 875. In one embodiment, as illustrated inFIG. 8D, the system may take the intersection of each expanded AOVs 875and the QOV 850 in order to locate associated terms 880 for query term Q801. In other embodiments, other functions may be used to locateassociated terms.

In certain embodiments, a similarity score between the query term Q andeach associated term may be calculated after associated terms for Q arelocated. The associated terms may then be ranked by their similarityscore values, so that the associated term with the highest similarityscore is ranked first. In certain embodiments, the similarity scorefunction may be a correlation coefficient distance measurement and itsvalue can be assigned to the resulting matching terms as a scoresignifying a final similarity measurement between the associated termand the initial query term, i.e., how much the results match the initialquery term.

In one embodiment, the similarity score between two vectors may becalculated by taking the sum of the relationship scores from theintersecting terms and multiplying it by the length of the vectorcomposed only of the intersecting terms. In another embodiment, thesimilarity score between two vectors may be a correlation coefficientdistance measurement function which uses the following equations:

${n\left( {\sum\limits_{i = 1}^{*}\left( {V\bigcap W} \right)_{k}} \right)}\mspace{14mu}{or}$${X}{\sum\limits_{i = 1}^{*}X_{k}}$

-   -   where        X=(V∩W)_(k)        -   V=query vector, and        -   W=any vector compared to the query vector.

In another embodiment, an uncentered Pearson correlation coefficientdistance measurement may be used to calculate the similarity scorebetween vectors of different sizes, wherein:

$r_{U} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}{\left( \frac{x_{i}}{\sigma_{x}^{(0)}} \right)\left( \frac{y_{i}}{\sigma_{y}^{(0)}} \right)}}}$where$\sigma_{x}^{(0)} = \sqrt{\frac{1}{n}{\sum\limits_{i = 1}^{n}x_{i}^{2}}}$

-   -   and wherein distance is defined by        d_(U)≡1−r_(U)

In certain embodiments, after the query result terms 880 are located,the vectors of each element returned for the query also extracted andcompared and scored for similarity. This step advantageously allows forthe results to be networked by intersecting the contents of theirvectors. The network created by the intersection may be used todetermine how the initial query results are related, in what contextthey relate, whether their connection is direct or indirect, and thestrength of their relationships.

The query result data and the relationship network built using that datamay thus advantageously show the relationship of the query term 801 toother terms, the relationship of vectors to one another, and thestrength of their relationships using a similarity score. In certainembodiments, the resulting relationship network of the query resultterms 880 and/or query-related vectors can be visualized if necessaryfor further interpretation. For example, FIG. 9 shows a graphvisualization 900 (not drawn to scale) for a relationship networkcreated in response to a query for the term “red.” Terms that have ahigher relationship score to the term “red” appear closer to “red,” suchas “cardinal” 654. Terms with a lower relationship score appear fartheraway, such as “paste” 655. A user may advantageously use a visualizationsimilar to FIG. 9 in order to quickly understand the relationshipbetween terms in the information database.

3. Example System Components

FIG. 10 illustrates a relationship network system 1000 according to oneembodiment. The relationship network system 1000 includes a web server1010 that generates and serves pages of a host web site to computingdevices 1020 of end users. Although depicted as desktop computers 1020,the computing devices 1020 may include a variety of other types ofdevices, such as cellular telephones and Personal Digital Assistants(PDA). The web server 1010 may be implemented as a single physicalserver or a collection of physical servers. Certain embodiments mayalternatively be embodied in another type of multi-user, interactivesystem, such as an interactive television system, an online servicesnetwork, or a telephone-based system in which users select items toacquire via telephone keypad entries and/or voice.

The web server 1010 provides user access to electronic informationrepresented within a database or a collection of databases 1020. Aninformation acquisition processor 1015 that runs on, or in associationwith, the web server provides functionality for users to enter a searchquery for information they would like to find. In one embodiment, theinformation represented in the database 1020 may include documents,characters, words, images, songs, or videos or any other data that maybe stored electronically. Many hundreds of thousands or millions ofbytes of data may be stored in the database.

In one embodiment, a document or other object in the informationdatabase 1020 may be retrieved using the information acquisitionprocessor 1015. Each object may be located by, for example, conducting asearch for the item via the information acquisition processor 1015, orby selecting the object from a browse tree listing.

As illustrated in FIG. 10, the relationship network system 1000 includesa relationship processor 1030 which is responsible for, among othertasks, creating relationship vectors for the data in the informationdatabase 1020. These relationship vectors are then stored in therelationships database 1040. In certain embodiments, the relationshipprocessor 1030 runs periodically and collectively analyzes or “mines”the information database in order to create and maintain therelationships database 1040 in response to new data that may be storedin the information database 1020.

In response to a query received by the information acquisition processor1015, the relationship network system 1000 sends the query to thenetwork generator 1050, which in addition to the query receivesrelationship vector information from the relationships database 1030 inorder to generate a relationship network based on the query. In certainrelationship network system embodiments, a set limit can be placed onthe number of relationships that are created in order to address thesubstantially large amounts of relationships that can be created in webspace, as discussed above.

The resulting relationship network is then sent to the query resultsprocessor 1060, which processes the results, optionally creates a visualrepresentation of the relationship network, and sends this data to theinformation acquisition processor 1015. The results data may then bereturned to computing devices 1020 that submitted the query via theInternet.

4. Example: Music Database

One embodiment of the invention may be implemented to discoverrelationships between human-generated content related to a database ofmusic. Some examples of human-generated content relating to music areplaylists, blogs, and recommendation lists. The system may determinerelationships between music files based on their location within adirectory or repository over a large data space, such as the Internet.This relationship data, which may include information such as theartist, album, title of the song and year of release, may be stored inassociative memory modules, and then be transferred into query objectvectors, as described above. Then, in response to a query, such as foran artist or a song, the system may create and present a relationshipnetwork of related artists or songs to the query and optionallyvisualize the relationship network.

5. Generating a Video Search Engine Results Page (VSERP)

FIG. 11 is a flow chart showing one embodiment of a process 1100 fordisplaying a VSERP. While this example is described with reference tovideo data, in other embodiments different types of media may beretrieved and displayed, such as audio data, images, and the like.

Initially, a user provides a query term or object at state 1101. Thisquery term, or search term, may be handled by a search module toretrieve at least one web page. The search module may be implemented inhardware, firmware, or software. The search module may provide thesearch results independently, or may use utilize other sources, such asan internet search engine, to return results. Based on the query object,a VSERP is created at state 1102 containing the ranked search resultsand ranked video content for each result. The creation of the VSERP maybe furthered by, at least, an analysis module responsible for generatinga content analysis of the retrieved search results using one or more ofthe methods described above. The content analysis may include additionalsearch terms for input into the search module or a specialized mediasearch module. The analysis module may be configured to provide anexecutive summary of a document, as described above. In this case, theanalysis module may comprise a summarization module configured toprovide a summary of a webpage. In order to achieve the functionality ofthe analysis module, the analysis module may additionally comprise aparser configured to parse a webpage into segments. The media searchmodule, which is functionally similar to the search module, retrievesmedia objects based on input search terms. A results compiler may serveto organize the information retrieved by the search module and mediasearch module and the information generated by analysis module. Theresults compiler may additionally be configured to display the organizedinformation. For instance, the results compiler may generate a web pagein HTML format for viewing on an internet browser. The results compilermay organize the information in a number of ways. In one embodiment, theresults compiler may generate a webpage with the retrieved media objectsembedded adjacent to the associated retrieved search result. At state1103, the VSERP is displayed. For example, a number of the most relevantsearch results may be displayed, along with one or more related videoobjects for each result. Relevant media objects may be returned in avariety of ways such as by displaying thumbnails and links on a searchengine results page. In other embodiments the videos may be playable onthe search engine results page. The videos or links to the videos mayalso be e-mailed, shared, displayed on a blog, or the like.

FIG. 12 is a flow chart of one embodiment of the process 1102 forcreating a VSERP. The process 1102 begins at state 1201 where aggregatedsearch results are returned. Aggregated search results may be obtainedby providing the query object of state 1101 to one or more internetsearch engines. For example, the query object may be processed by theGoogle, Yahoo!, and SeeqPod search engines, and the search result pagesfound by each of them may be retrieved and combined. In otherembodiments, a search is performed on an electronic database other thanthe internet. For example, a search may be performed on the GENEBANKgene database.

At state 1202, these aggregated search results are ranked. In certainembodiments, any result retrieved by multiple search engines is onlyincluded once in the ranked results. Rankings may be generated bymatching the context of each search result to the context of the query.This may be done according to the methods taught above, especially withreference to FIG. 8D. For example, a context for the search results maybe determined by constructing relationship vectors for each result. Thecontextually based relationship vector of the query term may beconstructed from pre-processed sources such as dictionaries,encyclopedias, on-line articles, and the like. The similarity scoresgenerated by comparing the query term's relationship vector to therelationship vectors of the search results may then be used to scoreeach search result. In other embodiments, the search results may beranked according to some other method, such as a keyword search rankingin which the document having the most occurrences of the query term isranked first.

At state 1203, each ranked search result is summarized. The process ofsummarizing the search results at state 1203 may include generatingrelationship vectors for each search result as described above. Tocreate a summary using these relationship vectors, each section of thepage is scored based upon the relationship scores of terms within thatsection. Sections of the page may include paragraphs, sentences,phrases, or the like. Scoring may be accomplished, for example, bysumming the relationship scores of a term's QOV, and then summing thesevalues for each term in a section. The highest scoring sections containhigh-value, contextually-associated words and make up a summary. Thisprocess is repeated for each search result.

At state 1204, each term from the summaries is extracted and used toform tags for its corresponding search result. At state 1205, the tagsextracted from the summary are expanded into multi-word phrases to beused in conjunction with the tags to provide better contextual matchingbetween the search results and media objects. Phrases may be valued morehighly than the individual terms that comprise them. For example, theterm ‘cat litter’ may be more relevant than the individual words ‘cat’and ‘litter’. Tags are expanded into multi-word phrases by combining thetags extracted from the summary with other highly relevant words. Theseother words may be obtained from a relationship network of the searchresult, a relationship network of a general knowledge database for thegiven tag, or some other source. Thus, an actual page may refer only tosand used in a cat hygiene product, but the phrase ‘cat litter’ maynonetheless be identified as an important multi-word phrase based upon ageneral relationship network. In other embodiments of the invention,state 1205 is optionally performed and the tags extracted from thesummaries at state 1204 may be used without creating multi-word phrases.

At state 1206, media storage sites are queried and media objectsreturned. The media storage sites may be any combination of electronicdatabases containing media data, such as a collection of video, image oraudio storage internet websites. For example, the media storage sitesmay comprise at least one of ITUNES, YOUTUBE, GOOGLE VIDEO, YAHOO!VIDEO, or the like. The sites are queried using the query objectprovided by the user at state 1101 of process 1100. A collection ofmedia objects related to the query object is retrieved for furtherprocessing, so that those media objects that are related to the searchresults may be identified. In other embodiments the system queries themedia storage sites using high value terms and phrases extracted from aresult page summaries, rather than the query object provided at state1101.

Video attributes are extracted at state 1207 of process 1102. Videoattributes are extracted by analyzing data from videos, images, speech,audio, and other contextual data surrounding any form of media. In otherembodiments, a collection of data associated with a video object isanalyzed to determine the most significant terms associated with thatobject and return these terms as extracted tags.

At state 1208, the extracted attributes are matched with the aggregatedsearch results. In order to match the results, the extracted attributesare compared with the summary and tag data generated at state 1205. Whenthe extracted attributes of a video are similar to or contextuallyrelated to a summary or tags, then the video is matched to the searchresult associated with the summary and tags. Video matches for eachsearch result are ranked according to the strength of the relationshipbetween the two objects. For example, the objects may be matched basedon the contextual similarity between each search result and media objectrelationship network. Relationship, vectors for the tags and multi-wordphrases for each search result may be compared to the relationshipvectors of the videos according to the process described above, forexample with respect to FIG. 8D. Thus, similarity scores may begenerated for the vectors. The multiple vectors associated with eachsearch result may be compared with the multiple vectors associated witheach video, and a quantitative representation of the strength of thematch may be derived from a function of the similarity scores.

While one embodiment of the system utilizes vector based relationshipmatching to retrieve media objects associated with a particular searchresult, embodiments of the invention are not limited to this particulartechnique. In other embodiments, matching a query result to a mediaobject may be based on a keyword search. For example, a keyword from thecontent or metadata of a search result may be used to perform asecondary search for media objects that relate to that keyword. Themedia objects that have the strongest association to the keyword canthen be displayed to the user on the VSERP.

In other embodiments, videos may be matched based at least in part onpreviously defined user matches. For example, a user may provide a queryterm and retrieve a VSERP. That user may then decide to manuallyassociate a particular media object he or she likes with one of thesearch results. The user may, for example, selecting a button on theVSERP to link a particular media object to a particular search result.This user-defined match may be stored in a database of such matches andlater accessed each time a search is performed. This mechanism allowsusers themselves to help determine which media objects should be moststrongly associated with a particular content page. In certainembodiments, user matched media objects will always appear withparticular search results. In other embodiments, user matching may be afactor in determining the relevance of a particular media object to asearch result.

Media objects matched to each result at state 1208 are ranked at state1209. The media objects are ranked based on the strength of theirrelationship to each search result. Thus, the highest ranked media for asearch result is the media most related to that search result. Thatmedia may in turn be ranked differently for other search results. In oneembodiment, the strength of the relationship between a media object anda search result is a function of the similarity scores of the differentvectors associated with each object. In some embodiments, the rankingmay be further dependent upon the source of the media and the theme ofthe search results. For example, a video from a sports news website maybe ranked higher than a video from a music video website depending onthe theme of the search result, regardless of the tags associated witheach video. Similarly, information as to the source of the searchresult, such as from a known sports news website, may be used in rankingthe videos in some embodiments.

In some embodiments, state 1209 further comprises re-ranking the searchresults based on a combination of the relevance of the search result tothe query term and the relevance of the media objects to the searchresult. For example, a first search result may initially be ranked asthe most relevant result at state 1202, and a second search result maybe ranked as the second most relevant result. However, the media objectsrelated to the first search result may have a low quantitativerelationship score with the first result. This score may be based, forexample, on the average score of three objects to be returned with thesearch result. In contrast, the second search result may have a highquantitative relationship score with its related media objects. Thesecond search result and its related media objects may therefore beranked above the first search result at state 1209.

The ranked results including the search results and the most relevantmedia objects for those results are returned as a VSERP at state 1210.

FIG. 13 is a flow chart showing a more detailed view of one embodimentof a process 1203 for generating summaries of documents returned assearch results. In other embodiments, this process may be used to createsummaries of objects in an electronic database that are independentlyuseful, without using those summaries to generate a VSERP. In certainembodiments, an electronic database may include, but is not limited to,a collection of characters or other forms of text, images, audio, video,or any other data that may be analyzed electronically. Objects or termswithin the electronic database may thus be documents, paragraphs,sentences, characters, words, images, songs, or videos.

In the embodiment illustrated, the system first obtains a search resultdocument to process at state 1301. For example, the document may be thehighest ranking document after the aggregated search results have beenranked. In other embodiments, the document selected may be chosen usinganother method, such as selecting documents alphabetically based ontheir titles, selecting documents based on size, or the like.

The system parses or normalizes the document at state 1302. In someembodiments, normalization includes extracting the plain text content,stopword removal, stemming, and filtering. Extracting the plain text mayinclude removing HTML syntax or the like. The process of stopwordremoval involves removing commonly occurring words that are of low value(e.g. a, of, as, the, on, etc.) so that the information databasecontains only valuable terms. Stemming replaces a word that is in aplural or verb form with its root. Filtering may include removing wordsfrom an undesired words list. While these processes have been describedwith respect to textual information, the invention is not limited totext-based data. Similar concepts may be applied to other types of data,for example media data, to create a narrowed document that contains onlyuseful information.

At state 1303 the system generates relationship vectors representing thedocument as described above. The relationship vectors are and used toscore sections of the document at state 1304 to determine the mostrelevant sections. The relationship vectors provide information as tothe relative uniqueness of terms and the relationship between terms,which may serve as a basis for scoring. Sections of an object containingmany terms that have a high relationship score or a high density of suchterms will in turn be scored highly. To determine which sections aremost relevant, the section scores are compared. In general, the sectionswith the highest scores will be the most relevant. In other embodiments,the section score may be further modified before ranking the sections.For example, the section score may be compared to the number of terms inthe section.

At state 1305 this scoring may be used to create a summary of thedocument by returning the most relevant sections to the user as asummary of the object. The sections may be returned in order ofrelevance, in the order they appear in the object, or based on someother factor. The system may also return a quantitative measure of therelevance of each section based on its section score. These summariesand scores may be used as described above to facilitate the matching ofsearch results to media objects. In other embodiments, the scoring datamay be used to generate recommended keywords or phrases for searchengine queries.

FIG. 14 is a flow chart showing more detail of one embodiment of theprocess 1304 for scoring sections of a document. At state 1203, asection that has not been scored is selected from the document. Sectionsmay be sentences, paragraphs, phrases, entire documents, or some otherportion of the object. For example, the document 400 shown in FIG. 4Amay have been returned as a search result for use in generating a VSERP.That document is now selected to be summarized in the current example.Since none of the sections have yet been scored, the first section isselected. In this example, sentences are used as sections and eachsentence is shown on a separate line. FIG. 15 shows a first sectionselected according to state 1401, and the first term 431 of that sectionaccording to state 1402. In this example, that term 431 is “bloom.”

Next, process 1304 scores the selected term at state 1403. The term isscored utilizing the relationship vectors generated in process 1203 atstate 1303. For example, the relationship vector may be a QOV having acore term identical to the selected term, such as the sample QOV 1600shown in FIG. 16. The QOV 1600 further includes a number of associatedterms 1602 with rankings 1601 based upon relationship scores 1603. Therelationship scores 1603 are used to calculate the term score. In oneembodiment, the relationship score for each of the associated terms issummed to provide the term score. Thus the term score of the core termrepresented by QOV 1600 would be S₁+S₂+ . . . +S_(N). In otherembodiments, sections may be scored using alternative methods. Forexample, information in an associated memory module may be used to formrelationship scores for terms without forming QOVs for those terms. Thesection score may also be determined according to an algorithm otherthan summing the relationship scores of the terms in the section. Forexample, the term frequency of a particular term across an object may becompared with the term frequency across a segment and also with thenumber of terms shared by the object and the section, and the resultingscore may be a function of these variables. In other embodiments,certain phrases are given higher scores than the sum of the term scoresthat comprise these phrases.

The term score is used at state 1404 to increment the section score.Initially, the section score is zero, and thus the new section scoreafter processing the first term will be equal to the term score.

Proceeding to decision block 1405, the system determines if the selectedterm is the last term in the selected section. If there are more terms,then the process 1304 returns to state 1402 and selects the nextunscored term 432 from the currently selected section. In the sampleshown in FIG. 15, that term 432 would be “March.” Process 1304 thenloops through states 1402, 1403, and 1404 until the last term 433 in theselected section is scored. In the example shown in FIG. 15, that term433 would be “fall.” For each term, a term score is calculated bysumming the relationship scores of the previously generated QOV for thatterm. The section score is incremented with each term, so that thesection score is the sum of all of the term scores for the terms in theselected section. When the last term is reached, process 1304 proceedsto decision block 1406.

At decision block 1406, the system determines if the selected section isthe last section in the document. If the section is not the last, thenthe process 1304 returns to state 11401 and selects the next unscoredsection from the document. In FIG. 15 that section 422 is the secondsentence which is represented on the second line. This newly selectedsection 422 proceeds through process 1304 in the same way as theprevious section, calculating the section score by summing the termscores, which are generated from the relationship scores. When the newsection is selected, it is associated with a new section score that isinitially zero. After every term in the section has been scored and thesection score is computed, the process 1304 returns to decision block1406. If the selected section is the final section in the object beinganalyzed, then process 1304 ends and process 1203 proceeds to state1305.

As an illustrative example of using the summary for a purpose other thancreating a VSERP, a database may contain a number of web pages returnedin an internet search. Each web page may contain several pages of text,making it impractical to review the entirety of each document. Further,a keyword search may highlight sections that do not embody the nature ofeach web page or display web pages using the same term in a differentcontext. A summary may therefore be desired for each result. For thefirst result, the summary generating system will score sections of thatweb page using the relationship vectors built from that document. Thesections of the first web page result are scored as described above, andthe highest scoring sections are returned as a summary. For example, thesections may be sentences and the top three scoring sentences may bereturned. Repeating the process on each returned search result, a userwould be able to quickly recognize the most relevant information frommany pages of material by reference to a number of three sentencesummaries. The summaries may be displayed, for example, along withsearch results on a search engine results page.

While the process for generating and displaying a VSERP is describedwith reference to the methods for generating relationship networksdescribed herein, it will be understood that other methods may be usedwithout departing from the scope of the invention. For example, certainembodiments of the current invention do not need to summarize theresults pages as described below. These embodiments may alternativelygenerate tags for search results using other methods. For example, tagsmay be defined by existing html tags or keywords for a search result. Inother embodiments, tags may be determined by finding commonly occurringterms within a search result document. These tags may in some cases befiltered to remove common words (e.g. ‘is’, ‘the’, ‘a’, etc.). Theprocess of matching also does not require the use of relationshipnetworks, and a variety of techniques may be utilized by the currentinvention. For example, media objects may be matched to search resultsbased entirely on a comparison of the tags generated for each, withoutuse of relationship vectors associated with each tag.

Conclusion

All of the features described above may be embodied in, and automatedby, software modules executed by general purpose computers. The softwaremodules may be stored in any type of computer storage device or medium.All combinations of the various embodiments and features describedherein fall within the scope of the present invention.

Although the various inventive features and services have been describedin terms of certain preferred embodiments, other embodiments that areapparent to those of ordinary skill in the art, including embodimentswhich do not provide all of the benefits and features set forth hereinand do not address all of the problems set forth herein, are also withinthe scope of this invention.

We claim:
 1. A method of searching for media objects associated with asearch term, the method being executed by a computer system andcomprising: receiving a first search term from a user; performing afirst search of Internet data, wherein the first search identifies webpages that include the first search term; analyzing a plurality ofpositional relationships between the first search term and additionalterms within an individual one of the web pages; scoring each of theplurality of positional relationships; determining, based on the scoringof each of the plurality of positional relationships, at least oneadditional search term; and performing a second search for media objectsrelating to the at least one additional search term.
 2. The method ofclaim 1, further comprising displaying a link to the identified webpages and a link to the media objects to the user.
 3. The method ofclaim 2, wherein a media object associated with a particular web page isdisplayed adjacent to a link to the particular web page.
 4. The methodof claim 1, wherein the at least one additional search term is aselected subset of the additional terms within the individual one of theweb pages.
 5. The method of claim 4, wherein the subset is selectedbased upon the scoring.
 6. The method of claim 1, wherein the scoring isbased upon a distance between the first search term and the additionalterms within the individual one of the web pages.
 7. The method of claim1, further comprising displaying a link to the identified web pages anddisplaying at least one media object returned by the second search. 8.The method of claim 1, wherein at least one of the media objectscomprises a video file, an audio file or an image file.
 9. The method ofclaim 1, wherein performing the first search of the Internet datacomprises using an internet search engine.
 10. The method of claim 1,further comprising summarizing, via the computer system, the web pages.11. The method of claim 1, wherein the analyzing comprises parsing, viathe computer system, the web pages.
 12. The method of claim 1, whereinat least one of the first search term and the additional search termscomprises a phrase having a plurality of words.
 13. A method ofretrieving media objects associated with a query, the method beingexecuted by a computer system and comprising: receiving a queryincluding a first search term from a user; retrieving, based upon thequery, at least one web page; generating a content analysis thatdescribes a plurality of positional relationships between the firstsearch term and terms within an individual one of the at least one webpage; and retrieving, based upon the content analysis, at least onemedia object, wherein the generating comprises scoring, via the computersystem, the plurality of positional relationships and the contentanalysis includes scoring information for the plurality of positionalrelationships.
 14. The method of claim 13, wherein the at least one webpage is retrieved using an internet search engine.
 15. The method ofclaim 13, wherein the scoring is based upon a distance between the firstsearch term and the terms within the individual one of the at least oneweb page.
 16. A non-transitory computer-readable storage medium storingmodules that, when executed by a processor, cause a computer to performa method of searching for media objects associated with a search term,the method comprising: retrieving, based upon a received first searchterm, at least one web page including the first search term; analyzing aplurality of positional relationships between the first search term andadditional terms within an individual one of the at least one web page;scoring each of the plurality of positional relationships; determining,based on the scoring, at least one additional search term, andretrieving, based on the at least one additional search term, at leastone media object.
 17. The non-transitory computer-readable storagemedium of claim 16, wherein the method further comprises displaying alink to the at least one web page and a link to the at least one mediaobject to a user.
 18. The non-transitory computer-readable storagemedium of claim 16, wherein the scoring is based upon a distance betweenthe first search term and the additional terms within the individual oneof the at least one web page.
 19. The non-transitory computer-readablestorage medium of claim 16, wherein the at least one media objectcomprises a video file, an audio file or an image file.
 20. Thenon-transitory computer-readable storage medium of claim 16, whereinsaid analyzing comprises summarizing the at least one web page.
 21. Thenon-transitory computer-readable storage medium of claim 16, wherein theanalyzing comprises parsing the at least one web page.
 22. Thenon-transitory computer-readable storage medium of claim 16, wherein theanalyzing comprises generating a plurality of relationship vectorsconfigured to represent relationships between the first search term andthe additional terms.
 23. The non-transitory computer-readable storagemedium of claim 16, wherein the retrieving the at least one web pageretrieving comprises accessing an internet search engine.
 24. Anon-transitory computer-readable storage medium storing modules that,when executed by a processor, cause a computer to perform a method ofsearching for media objects associated with a search term, the methodcomprising: receiving a first search term from a user; performing afirst search of Internet data, wherein the first search identifies webpages including the first search term; analyzing a plurality ofpositional relationships between the first search term and additionalterms within an individual one of the web pages; scoring the pluralityof positional relationships; determining, based on the scoring theplurality of positional relationships, at least one additional searchterm; and performing a second search for media objects relating to theat least one additional search term.
 25. The non-transitorycomputer-readable storage medium of claim 24, where the scoring is basedupon a distance between the first search term and the additional termswithin the individual one of the web pages.